Repository navigation
Report the pipelines test wait timeout when it fails a read before it is sent - #7820
Merged
Merged
Conversation
… is sent The SDK rate limiter fails a request whose context is already canceled with a bare "context canceled", so a timeout that passed between two reads surfaced as "failed to fetch pipeline events: failed in rate limiter: context canceled" instead of the wait timeout. Return the context's cause for a read that fails after the wait was canceled. Co-authored-by: Isaac <no-reply@databricks.com>
… check Only add the wait's cause to a context-canceled read error, wrapping it so the read's own message stays, and keep the caller's context.Cause check as a fallback for any read path that does not go through waitReadError. Co-authored-by: Isaac <no-reply@databricks.com>
andrewnester
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
In
waitForTestUpdate(databricks pipelines test), a read that fails withcontext.Cancelednow also carries the wait context's cause (errTestWaitTimeout). The read's own error stays wrapped. Other read errors are returned unchanged, even if the context was canceled by then. The command's caller still checkscontext.Causeas a fallback.Why
TestWaitForTestUpdateTimeoutfailed once on main on Windows:The 20ms timeout can pass between two reads. Windows timers are coarse, so this happens there. A request that is already in flight gets the cancel cause from
net/http. But the SDK rate limiter checks the context before sending, and it fails with a barecontext canceled, which hides the timeout. The command itself reported this case correctly, because it checkedcontext.Causeseparately. The function didn't, so the test was flaky.This is not a regression from #7800: that PR changed
pipelines runandbundle run(cmd/pipelines/run.go,bundle/run/). It did not changepipelines test's wait loop (cmd/pipelines/test.go,test_events.go). The flaky test came with #7426.Tests
New
TestWaitReadErrorcovers: a real error is kept, and a context error gets the cause and keeps its message. NewTestWaitForTestUpdateTimeoutBeforeRead: the wait context is already canceled with the timeout cause, so the first read fails in the rate limiter. Without the fix it fails with the same message as on CI; with the fix it passes. The otherTestWaitForTestUpdate*tests and thepipelinesacceptance tests pass.Decision: this does not change one existing behavior. If the timeout fires during the final events read after the update has already ended, the run is still reported as timed out. That is unchanged from main and rare, since it needs the timeout to hit that exact read.
Dashboard
Target error should be in err chain: expected "test wait timed out", in chain: "failed to fetch pipeline events: failed in rate limiter: context canceled"Test: cmd/pipelines.TestWaitForTestUpdateTimeout
This pull request and its description were written by Isaac.